Incremental Prediction of Sentence-final Verbs: Humans versus Machines

Verb prediction is important in human sentence processing and, practically, in simultaneous machine translation. In verb-final languages, speakers select the final verb before it is uttered, and listeners predict it before it is uttered. Simultaneous interpreters must do the same to translate in real-time. Motivated by the problem of SOV-SVO simultaneous machine translation, we provide a study of incremental verb prediction in verb-final languages. As a basis of comparison, we examine incremental verb prediction with human participants in a multiple choice setting using crowdsourcing to gain insight into incremental human performance in a constrained setting. We then examine a computational approach to incremental verb prediction using discriminative classification with shallow features. Both humans and machines predict verbs more accurately as more of a sentence becomes available, and case markers—when available—help humans and sometimes machines predict final verbs. 1 The Importance of Verb Prediction Humans predict future linguistic input before it is observed (Kutas et al., 2011). This predictability has been formalized in information theory (Shannon, 1948)—the more predictable a word is, the lower the entropy—and has explained various linguistic phenomena, such as garden path ambiguity (Den and Inoue, 1997; Hale, 2001). Such instances of linguistic prediction are fundamental to statistical NLP. Auto-complete from search engines has made next-word prediction one of best known NLP applications. Long-distance word prediction, such as verb prediction in SOV languages (Levy and Keller, 2013; Momma et al., 2015; Chow et al., 2015), is important in simultaneous machine translation from subject-object-verb (SOV) languages to subjectverb-object (SVO) languages. In SVO languages such as English, for example, the main verb phrase usually comes after the first noun phrase—the main subject—in a sentence, while in verb-final languages such as Japanese or German, it comes very last. Human simultaneous translators must make predictions about the unspoken final verb to incrementally translate the sentence. Minimizing interpretation delay thus requires making constant predictions and deciding when to trust those predictions and commit to translating in real-time. Such prediction can also aid machines. Matsubara et al. (2000) use pattern-matching rules; Grissom II et al. (2014) use a statistical n-gram approach; and Oda et al. (2015) extend the idea of using prediction by predicting entire syntactic constituents for English-Japanese translation. These systems require fast, accurate verb prediction to further improve simultaneous translation systems. We focus on verb prediction in verb-final languages such as Japanese with this motivation in mind. In Section 2, we present what is, to our knowledge, the first study of humans’ ability to incrementally predict the verbs in Japanese. We use these human data as a yardstick to which to compare computational incremental verb prediction. Incorporating some of the key insights from our human study into a discriminative model—namely, the importance of case markers— Section 3 presents a better incremental verb classifier than existing verb prediction schemes. Having established both human and computer performance on this challenging and interesting task, Section 4 reviews our work’s relationship to other studies in NLP and linguistics. 2 Human Verb Prediction We first examine human verb selection in a constrained setting to better understand what performance we should demand of computational approaches. While we know that humans make incremental predictions across sentences, we do not know how skilled they are in doing so. While it’s possible that machines—with unbounded memory and access to Internet-sized data—could do better than humans, this study allows us to appropriately gauge our expectations for computational systems. We use crowdsourcing to measure how well novice humans can predict the final verb phrase of incomplete Japanese sentences in a multiple choice setting. We use Japanese text of the Kyoto Free Translation Task corpus (Neubig, 2011, KFT), a collection of Wikipedia articles in English and Japanese, representing standard, grammatical text and readily usable for future SOV-SVO machine translation experiments. 2.1 Extracting Verbs and Sentences This section describes the data sources, preparation, and methodology for crowdsourced verb prediction. Given an incomplete sentence, participants select a sentence-final verb phrase containing a verb from a list of four choices to complete the sentence, one of which is the original completion. We randomly select 200 sentences from the development set of the KFT corpus (Neubig, 2011). We use these data because the sentences are from Wikipedia articles and thus represent widely-read, grammatical sentences. These data are directly comparable to our computational experiments and readily usable for future SOV-SVO machine translation experiments. We ask participants to predict a “verb chunk” that would be natural for humans. More technically, this is a sentence-final bunsetsu.1 We identify verb bunsetsu with a dependency parser (Kurohashi and Nagao, 1994). Of interest are bunsetsu at the end of a sentence that contain a verb. We also use bunsetsu for segmenting the incomplete sentences we show to humans, only segmenting between bunsetsu to ensure each segment is a meaningful unit. A bunsetsu is a commonly used linguistic unit in Japanese, roughly equivalent to an English phrase: a collection of content words and zero or more functional words. Japanese verb bunsetsu often encompass complex conjugation. For example, a verb phrase 読みたくなかった (read-DESI-NEG-PAST), meaning ‘didn’t want to read’, has multiple tokens capturing tense, negation, etc. necessary for translation. Answer Choice Selection We display the correct verb bunsetsu and three incorrect bunsetsu completions as choices that occur in the data with frequency close to the correct answer in the overall corpus. We manually inspect the incorrect answers to ensure that these choices are semantically distant, i.e., excluding synonyms or troponyms. Sentence Presentation We create two test sets of truncated sentences from the KFT corpus: The first, the full context set, includes all but the final bunsetsu—i.e., the verb phrase—to guess. The second set, the random length set, contains the same sentences truncated at predetermined, random bunsetsu boundaries. The average sentence length is nine bunsetsu, with a maximum of fourteen and minimum of three. We display sentences in the original Japanese script. Participants view the task as a game of guessing the final verb. Each fragment has four concurrently displayed completion options, as in the prompt (2) and answers (3). Users receive no feedback from the interface. We use CrowdFlower2 to collect participants’ answers, at a total cost of approximately USD$300. From an initial pool of fifty-six participants, we remove twenty via a Japanese fluency screening. We verify the efficacy of this test with non-native but highly proficient Japanese learners; none passed. We collect five judgments per sentence from each participant. (2) 谷崎潤一郎は Junichiro Tanizaki-TOP